Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Fix controller-tools doesn't support single files as input #864

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

yongxiu
Copy link

@yongxiu yongxiu commented Nov 30, 2023

Fix #837

Below command

go run ../vendor/sigs.k8s.io/controller-tools/cmd/controller-gen webhook paths=./apis/cluster_webhook.go output:webhook:artifacts:config=./webhook/

Used to work, but didn't work any more since v0.9.0, which is caused by #663. There is a assumption we want the base part of the path to be either "..." or ".", this is untrue if user specified a specific file, I added a logic to exclude golang file in this case.

Another logic is uniquePkgIDs, as from comment there is not harm that comes from this per se, but it makes testing easier when a known number of modules can be asserted, this is fine however if there are two files coming from same package, second file will be ignored, thus I have to move this logic to test file.

Copy link

linux-foundation-easycla bot commented Nov 30, 2023

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: yongxiu / name: yongxiu (dabf09d)

@k8s-ci-robot k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Nov 30, 2023
@k8s-ci-robot
Copy link
Contributor

Welcome @yongxiu!

It looks like this is your first PR to kubernetes-sigs/controller-tools 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/controller-tools has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot
Copy link
Contributor

Hi @yongxiu. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 30, 2023
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: yongxiu
Once this PR has been reviewed and has the lgtm label, please assign sbueringer for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Nov 30, 2023
Copy link

@rmohr rmohr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yongxiu would you mind adding some context on how you are solving this to the PR description and/or commit?

@yongxiu
Copy link
Author

yongxiu commented Nov 30, 2023

@yongxiu would you mind adding some context on how you are solving this to the PR description and/or commit?

Done

@yongxiu yongxiu requested a review from rmohr November 30, 2023 17:12
@rmohr
Copy link

rmohr commented Nov 30, 2023

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 30, 2023
@vincepri
Copy link
Member

Why not specify the entire folder even if there is a single file in it?

@vincepri
Copy link
Member

paths is a list of directories, not files

Copy link

@rmohr rmohr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

lgtom to me, but requires a project owner to have a look :)

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 30, 2023
Copy link

@rmohr rmohr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/remove-lgtm

based on:

paths is a list of directories, not files

waiting for the out come of the discussion.

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 30, 2023
@yongxiu
Copy link
Author

yongxiu commented Nov 30, 2023

Why not specify the entire folder even if there is a single file in it?

There could be multiple files.

├── apis
│   ├── cluster_webhook.go
│   ├── machine_webhook.go
│   ├── typeBCluster_webhook.go
│   ├── typeBMachine_webhook.go

In above case, I want to generate webhook for cluster_webhook and machine_webhook in output directory A, while typeBCluster_webhook and typeBMachine_webhook to output directory B.

go run ../vendor/sigs.k8s.io/controller-tools/cmd/controller-gen webhook paths=./apis/cluster_webhook.go paths=./apis/machine_webhook.go  output:webhook:artifacts:config=./webhook/normal/
go run ../vendor/sigs.k8s.io/controller-tools/cmd/controller-gen webhook paths=./apis/typeBCluster_webhook.go paths=./apis/typeBMachine_webhook.go output:webhook:artifacts:config=./webhook/special/

paths is a list of directories, not files

While, at least the command line example is using single file as well:

controller-gen object paths=./apis/v1beta1/some_types.go

@rmohr
Copy link

rmohr commented Nov 30, 2023

@vincepri being able to separate output directories like described above sounds useful and it seemed to work before. Is there a clear idea if that should only be dirs, or is it more an interpretation of the word path that it should be a directory?

@joelanford
Copy link
Member

@yongxiu, can you add a test that covers using a single file within a directory? The test should probably cover:

  1. Generation does not occur for types in other files in the directory.
  2. If the file that we're generating for references identifiers (types, functions, variables, etc.) in this package, but in another file in this package, generation still works and includes those referenced items as necessary.

My hunch is that the reason that the current code doesn't accept individual files is because the Go AST parser operates on package directories, which makes handling my item (2) above trivial.

@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 7, 2023
@yongxiu
Copy link
Author

yongxiu commented Dec 7, 2023

Added webhook integration tests as requested, please take another look, thanks. @joelanford

@joelanford
Copy link
Member

joelanford commented Dec 8, 2023

Thanks for adding the webhook integration test. That looks good. Can you also add similar tests that cover CRD and RBAC generation? I assume the changes in loader.go apply to all generation not just webhooks, correct?

I'm particularly interested in CRDs with my item (2). If a CRD's types are split between multiple files (say widget_types.go and shared.go) where widget_types.go references types in shared.go that are necessary for the CRD schema generation. What happens when widget_types.go is passed in to the loader?

@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 8, 2023
@yongxiu
Copy link
Author

yongxiu commented Dec 8, 2023

@joelanford, added tests for CRDs and rbac as well

For the case you mentioned, I have this case in the test, it needs to pass in both files, otherwise it will only shows type: object for that field without detail fields, and if we run controller-tool cmd, it will give error:

$ go run cmd/controller-gen/main.go  crd paths=pkg/crd/testdata/multiple_files/file_two.go output:dir=pkg/crd/testdata/
multiple_files/
controller-tools/pkg/crd/testdata/multiple_files/file_two.go:31:7: unknown type TwoResourceSpec
Error: not all generators ran successfully

To clarify, I also tested controller-tools v0.8.0, it give the same error.

@joelanford
Copy link
Member

joelanford commented Dec 11, 2023

Not suggesting you go and do this before we see what other maintainers think, but I wonder if we could:

  • accept a single file as the path
  • load the entire package directory that contains the file
  • detect types that are defined in the specified file and only generate based on those types

Alternately, another potential option is leaving paths supporting only package directories, but then add other option that lets a caller pass in a specific list of types they want to generate for?

@yongxiu
Copy link
Author

yongxiu commented Dec 13, 2023

IMHO, detecting the types or add one more input of types is over kill. This feature has been broken for 5 releases, which is 1.5 years, and there is only one bug: #837 for this. I don't see much value of adding types, accepting multiple files which is changing back to 0.8.0 behavior is good enough for most of developers.

@yongxiu
Copy link
Author

yongxiu commented Dec 19, 2023

I wonder how can we move this forward? Are we waiting for any other stakeholders?

@yongxiu
Copy link
Author

yongxiu commented Jan 30, 2024

Friendly ping!

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 16, 2024
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 29, 2024
@k8s-ci-robot
Copy link
Contributor

@yongxiu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-controller-tools-test-master dabf09d link true /test pull-controller-tools-test-master

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 15, 2024
@k8s-ci-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@airycanon
Copy link

@yongxiu @joelanford Hi, Is there any progress on this PR?
My project relies on the single files input, and to use it, I have to keep my Go version at 1.21, as the old version that includes this feature causes a panic when running on Go 1.22.

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle stale
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 14, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle rotten
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Sep 14, 2024
@rmohr
Copy link

rmohr commented Sep 16, 2024

@joelanford we would still be interested in that change. Would you have some recommendations on how to move this forward?

@rmohr

This comment was marked as duplicate.

@rmohr
Copy link

rmohr commented Sep 16, 2024

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Sep 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

controller-gen fails to handle individual file in paths
7 participants